Project: Finding Lane Lines on the Road

---

The main goal of this notebook is to use computer vision techniques in order to identify lanes in a variety of images and later on videos.

Here is one example of an image before and after being pre-processed.

This is the first asigment of the Self-Driving Car Engineer Nanodegree Program from Udacity

Pipeline

---

• Import and standarize images
• Enhance colors with hsl color palette
• Convert to Gray scale
• Gaussian smoothing.
• Canny edge detector
• Region of interest segmentation
• Hough transform
• Draw straight lines
  • Categorize lines by slope and x position
  • Apply unidimensional fitting
  • Obtain extreme points from the fitting equation
  • Extrapolate to fill image
• Overlap lines with original image
• For videos:
  • Apply a runnign average filter for frames
  • filtero out sudden changes in lines between frames

Import Base Packages

#importing some useful packages
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import os
import math
import numpy as np
import cv2
%matplotlib inline

Image size homogenization

---

Pretty simple function that resizes the images so all images have a standarized size for the pipeline of x=960 px by y=540 px. It is specially usefull when the boundaries of the regions of interest are defined by pixels.

Be aware that the origin of the images is in the upper right corner. Think of them as matrices not as images.

[function] Resize Images

def resizeImage(image):
    """
    image should be a np.array image, note that it will be modified
    """
    ysize = image.shape[0]
    xsize = image.shape[1]

    # Resize image if necesaary
    if xsize != 960 and ysize != 540:
        image = cv2.resize(image, (960, 540),)

    return image

Import images

Preview images to analize before start working on them

# Read list of images from a folder and obtain path
images_path ='test_images/'
names = os.listdir(images_path)
images_list = [images_path + item for item in names]
#Create matplotlib images from a list of paths
images =  [mpimg.imread(item, cv2.IMREAD_COLOR) for item in images_list]; 

[function] Display a list of images

def displayListImages(images,titles,cols=1,rows=1,):
    """
    Function to display and resize a list of images
    images is a list of matplotlib image (use imread)
    titles is a list of strings
    cols is an integer with the number of columns to display
    rows is an integer with the number of rows to display
    """

    # Helper to adapt images to full width
    plt.rcParams['figure.figsize'] = [12, 4*rows]
    plt.rcParams['figure.dpi'] = 100 # 200 e.g. is really fine, but slower

    for i in range(len(images)):
        
        plt.subplot(rows, cols, i+1)
        
        image = resizeImage(images[i])
        
        plt.title(titles[i]) 
        plt.imshow(image, cmap=None)

        
#Test function
displayListImages(images,cols=2,rows=6,titles=names)

Enhance and filter image colors

---

Decent results can be obtained just in the RGB space but for a more robust approach, color enhancement is very effective.

A color enhancement is very usefull especially when looking for yellow lines, hence different techniques will be tested in order to find the best color filter to obtain the lines, specially under different tipes of tarmac or with the pressence of shades.

Isolate RGB color layers in the different color spaces: RGB, HSL, HSV to see wich ones contain onfo about line lanes.

The image below ilustrates de RGB, HSV and HSL color

In order to develop the color filters later on here the image below can be found with some explanation

*Important:*. Be aware that the images to be able to be processed and filtered corrected should be 8 bit images that is achieved using the following parameter when importing the image into a numpy array: mpimg.imread(image_file_path, cv2.IMREAD_COLOR)

---

First of all we are going to detect wich color space has got more relevant features. For that we are going to use the the cell below how each color space transfroms the image and how much information is cointained in each color space plane by filter them induavidually.

imagesCEP = []
titlesCEP = []
# Use a tricky image to see wich color enhanment is best suited
imageTest =  mpimg.imread(images_list[1], cv2.IMREAD_COLOR)
imageTest = resizeImage(imageTest)
#!!!!cv2.IMREAD_COLOR is crucial to be set so the image gets imported as 8 bits instead of floating point

# Change to different color spaces
hls = cv2.cvtColor(imageTest, cv2.COLOR_BGR2HLS)
hsv = cv2.cvtColor(imageTest, cv2.COLOR_BGR2HSV)

#rgb
imagesCEP.append(imageTest);titlesCEP.append("RGB");
imagesCEP.append(hls);titlesCEP.append("HLS");
imagesCEP.append(hsv);titlesCEP.append("HSV");

#rgb
imagesCEP.append(imageTest[:,:,0]);titlesCEP.append("imageR");
imagesCEP.append(imageTest[:,:,1]);titlesCEP.append("imageG");
imagesCEP.append(imageTest[:,:,2]);titlesCEP.append("imageB");

#hsl
imagesCEP.append(hls[:,:,0]);titlesCEP.append("hslH");
imagesCEP.append(hls[:,:,1]);titlesCEP.append("hslL");
imagesCEP.append(hls[:,:,2]);titlesCEP.append("hslS");

#hsv
imagesCEP.append(hsv[:,:,0]);titlesCEP.append("hsvH");
imagesCEP.append(hsv[:,:,1]);titlesCEP.append("hsvS");
imagesCEP.append(hsv[:,:,2]);titlesCEP.append("hsvV");


displayListImages(imagesCEP,titles=titlesCEP,cols=3,rows=4,)

Tip: If the features that we are interested in cannot be seen in one of the images below that means that that plane has got a lot of information about that feature and does not blend in with the rest of the image.

As it is possible to see in the iimage above the hsl color spaces seem to do a beter job finding yellow lines than the rbg space. Even the hsl and the hsv are prety similar, specially with the hue value where most of the information about the yellow line is contained, we are going to use the hsl as it becomes trivial to look for white lines.

---

Once the desired color space has been selected the next step is to develop a filter for the white pixels and another filter for yellow pixels then merge both with an or operation and then apply them to the original image with an and operation. Here is another good article on color filtering

In order to tune the filters, the l and s values are quite logical and work as expected in their range of 8 bits (0-255), but for the parameter h that should be in the range 0-180 (h=h/2) to make it fit in 8 bits I could not find a suitable map that correlates to the inputed values so it was basically trial and error.

# Cell to test color isolation
imageshsl = []
titleshsl = []

hls = cv2.cvtColor(imageTest, cv2.COLOR_BGR2HLS)

#find lleyow
# I think h is inverted as the H values shoudl be between 20 and 40 degrees for yellow and the range of the function is [0-180]
color1_hls = (80, 100, 0)
color2_hls = (100, 255, 255)
#Range HLS [0-180,0-254,0-254]

mask1 = cv2.inRange(hls, color1_hls,color2_hls)
imageshsl.append(mask1);titleshsl.append("mask Yellow");

#find white
color1_hls_w = (0, 200, 0)
color2_hls_w = (180, 255, 255)
#Range HLS [0-180,0-254,0-254]

mask2 = cv2.inRange(hls, color1_hls_w,color2_hls_w)
cv2.imwrite("mask2.png", mask2)

imageshsl.append(mask2);titleshsl.append("mask White");

#    Add masks together
mask = cv2.bitwise_or(mask1, mask2)

#Apply mask to target
res = cv2.bitwise_and(imageTest,imageTest, mask= mask)

#plt.imshow(mask, cmap='gray')   # this colormap will display in black / white
#plt.show()
#res = cv2.cvtColor(res, cv2.COLOR_BGR2GRAY)
imageshsl.append(res);titleshsl.append("res");

displayListImages(imageshsl,titles=titleshsl,cols=2,rows=2,)

As it can be seen the end result is pretty cool, in wich all the backgorund noise has been filtered out and basically just the line lanes and a couple of other features stand out.

Tip: Color enhancement is specially usefull when there are changes in the backgorund as tarmac varaitions or shadows as its able to filter out most of that noisy backgorund.

---

Note: Open cv uses BGR instead of RGB color channel by default.

[function] Color enhancement

def colorEnhancement(img):
    """Converts the image to HSL
    Creates two masks to filter white and
    lleyow lines
    Applies the mask
    Be carefull to input an 8bit image!
    cv2.IMREAD_COLOR is your friend when using imgread
    """
    hls = cv2.cvtColor(img, cv2.COLOR_BGR2HLS)
    ##find lleyows
    color1_hls = (80, 100, 0)
    color2_hls = (100, 255, 255)
    mask1 = cv2.inRange(hls, color1_hls,color2_hls)
    
    #find whites
    color1_hls_w = (0, 200, 0)
    color2_hls_w = (180, 255, 255)
    mask2 = cv2.inRange(hls, color1_hls_w,color2_hls_w)
    
    #    Add masks together
    mask = cv2.bitwise_or(mask1, mask2)
    
    res = cv2.bitwise_and(img,img, mask= mask)
    return res

#Test function
imagesCE=[]
titlesCE=[]
imageCE = colorEnhancement(imageTest)

imagesCE.append(imageTest);titlesCE.append("Original");
imagesCE.append(imageCE);titlesCE.append("Color enhanced");
    
displayListImages(imagesCE,cols=2,rows=1,titles=titlesCE)

Grayscale

---

As its name states, converts the 3 channel HSL or RGB (or even RGBA) to a single channel image. Even some information is lost on the way, it is very helpfull as it speeds up computation

[function] Grayscale

def grayscale(img):
    """Applies the Grayscale transform
    This will return an image with only one color channel
    but NOTE: to see the returned image as grayscale
    (assuming your grayscaled image is called 'gray')
    you should call plt.imshow(gray, cmap='gray')"""
    return cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    # Or use BGR2GRAY if you read an image with cv2.imread()
    # return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

#Test function
imagesGray=[]
titlesGray=[]
imageGray = grayscale(imageCE)

imagesGray.append(imageCE);titlesGray.append("Original");
imagesGray.append(imageGray);titlesGray.append("Gray scaled");
    
displayListImages(imagesGray,cols=2,rows=1,titles=titlesGray)

The end result is not very different (espceially when plotted in RGB), but computationally is going to be a good improvement.

Gaussian blur

---

Basically the functions smoothes out the image and takes out sharp edges. It works by looping through all the pixels of the image with a matrix of size defined as the _kernelsize and averages out the pixels in each region.

[function] gaussian_blur

def gaussian_blur(img, kernel_size):
    """Applies a Gaussian Noise kernel"""
    return cv2.GaussianBlur(img, (kernel_size, kernel_size), 0)

#Test function
imagesGauss=[]
titlesGauss=[]
imageGauss = gaussian_blur(imageGray,5)

imagesGauss.append(imageGray);titlesGauss.append("Original");
imagesGauss.append(imageGauss);titlesGauss.append("Gaussian blurr. Kernel=5");
    
displayListImages(imagesGauss,cols=2,rows=1,titles=titlesGauss)

Canny edge detection

---

Lets see if we can make it easier for the computer to find the lines if we apply an edge detector. It is a very interesting detector that works with two thresholds.

It basically looks for gradient changes along the edges detected afted applying a 5x5 Gaussian filter.

After, it analices the full image by discarding the isloated gradient changes detected as edges,

Then uses two thresholds to make the edge neat, everything below the lower threshold is discarded. Everything above the high threshold is retaine. De data in between the lower and the upper is retained if continuous contours can be found. More info here

[function] cannyEdge

def cannyEdge(img, low_threshold=250,high_threshold=450):
    imgCanny = cv2.Canny(img, low_threshold, high_threshold)
    return imgCanny

#Test function
imagesCanny=[]
titlesCanny=[]
imageCanny = cannyEdge(imageGauss,250,450)

imagesCanny.append(imageGauss);titlesCanny.append("Original");
imagesCanny.append(imageCanny);titlesCanny.append("Canny edge. low th=250.  high th=450");
    
displayListImages(imagesCanny,cols=2,rows=1,titles=titlesCanny)

Region of interest

---

Until now some filters have been applied but they are not enough to isolate just the lines from the rest of the picture, so some image masking for the area with the relevant features is required.

The Region of interes selected is a four side polygon wich can be shaped in width and height.

[function] region_of_interest

def region_of_interest(img, vertices):
    """
    Applies an image mask.
    
    Only keeps the region of the image defined by the polygon
    formed from `vertices`. The rest of the image is set to black.
    `vertices` should be a numpy array of integer points.
    """
    #defining a blank mask to start with
    mask = np.zeros_like(img)   
    
    #defining a 3 channel or 1 channel color to fill the mask with depending on the input image
    if len(img.shape) > 2:
        channel_count = img.shape[2]  # i.e. 3 or 4 depending on your image
        ignore_mask_color = (255,) * channel_count
    else:
        ignore_mask_color = 255
        
    #filling pixels inside the polygon defined by "vertices" with the fill color    
    cv2.fillPoly(mask, vertices, ignore_mask_color)
    
    #returning the image only where mask pixels are nonzero
    masked_image = cv2.bitwise_and(img, mask)
    return masked_image

#Test function
imagesROI=[]
titlesROI=[]

ysize =imageCanny.shape[0]
xsize =imageCanny.shape[1]

ROI_upperWidth = 100 #Width of the upper horizontal straight in px
ROI_upperHeight = 200 #Height of the upper horizontal straight from the bottom of the image in px
ROI_lowerWidth = 750 #Width of the lower horizontal straight in px
ROI_lowerHeight = 50  #Height of the lower horizontal straight  from the bottom of the image in px      
    
limitLL = ((xsize/2)-(ROI_lowerWidth/2),ysize-ROI_lowerHeight);
limitLR = (xsize - ((xsize/2)-(ROI_lowerWidth/2)),ysize-ROI_lowerHeight);
limitUL = ((xsize/2)-(ROI_upperWidth/2), ysize-ROI_upperHeight);
limitUR = ((xsize/2)+(ROI_upperWidth/2), ysize-ROI_upperHeight);
vertices = np.array([[limitLL,limitUL,limitUR , limitLR]], dtype=np.int32)

imageROI = region_of_interest(imageCanny,vertices)

imagesROI.append(imageCanny);titlesROI.append("Original");
imagesROI.append(imageROI);titlesROI.append("ROI");
    
displayListImages(imagesROI,cols=2,rows=1,titles=titlesROI)

# Plot vertices for verification in original image
x = [limitLL[0], limitUL[0], limitUR[0], limitLR[0],limitLL[0]]
y = [limitLL[1], limitUL[1], limitUR[1], limitLR[1],limitLL[1]]
plt.plot(x, y, 'b--', lw=4)

[<matplotlib.lines.Line2D at 0x1cfc6a6aa20>]

Hough transform

---

Even though it is pretty clear where the lines are, they are still in image language, so we need to extract the analytical features of the lines, that mean position and slope on the image. For that the hough transform comes in handy

[function] hough_lines

def weighted_img(img, initial_img, α=0.8, β=1., γ=0.):
    """
    `img` is the output of the hough_lines(), An image with lines drawn on it.
    Should be a blank image (all black) with lines drawn on it.
    
    `initial_img` should be the image before any processing.
    
    The result image is computed as follows:
    
    initial_img * α + img * β + γ
    NOTE: initial_img and img must be the same shape!
    """
    return cv2.addWeighted(initial_img, α, img, β, γ)


def draw_lines(img, lines, color=[255, 0, 0], thickness=2):
    """
    This function draws `lines` with `color` and `thickness`.    
    Lines are drawn on the image inplace (mutates the image).
    If you want to make the lines semi-transparent, think about combining
    this function with the weighted_img() function below
    """ 
    for line in lines:
        #print(line)
        for x1,y1,x2,y2 in line:
            cv2.line(img, (x1, y1), (x2, y2), color, thickness)
    

                
            
def hough_lines(img, rho, theta, threshold, min_line_len, max_line_gap):
    """
    `img` should be a np.array image
    Returns an image with hough lines drawn and the hough lines points.
    """
    lines = cv2.HoughLinesP(img, rho, theta, threshold, np.array([]), minLineLength=min_line_len, maxLineGap=max_line_gap)
    line_img = np.zeros((img.shape[0], img.shape[1], 3), dtype=np.uint8)

    
    try:
        draw_lines(line_img, lines, color=[255, 0, 0], thickness=4)
    except:
        lines = []
        print("No line found")
        pass
    return line_img,lines

    


#Test function
imagesHough=[]
titlesHough=[]

# Hough parameters
rho = 1 # distance resolution in pixels of the Hough grid
theta = 1*np.pi/180 # angular resolution in radians of the Hough grid
threshold = 1    # minimum number of votes (intersections in Hough grid cell)
min_line_len = 5 #minimum number of pixels making up a line
max_line_gap = 40   # maximum gap in pixels between connectable line segments

imgHouhg,lines = hough_lines(imageROI, rho, theta, threshold, min_line_len, max_line_gap,)

imagesHough.append(imageROI);titlesHough.append("Original");
imagesHough.append(imgHouhg);titlesHough.append("Hough transform");
    

displayListImages(imagesHough,cols=2,rows=1,titles=titlesHough)
#print(lines)

Draw straight lines

---

Now that the points of the candidates for the line lanes are known, the next logical step is to draw those lines.

For that first a categorization is done based on the slope of the lines and the position in the screen. There is also a thresholdSlope to eliminate unwanted too shallow angled straights detected as lines.

Then a one dimensional fit algorithm is applied to extract the parameters of the best fitting straigh given by the categorized points.

Finally the equation of the straight y=mx+b is employed to extrapolate the points. As is more aesthetically visual that the lines have the same height the y points are fixed and the x points are calculated instead, inverting the equation: x=1/m -b/m.

---

Note: An axis transsformation is required when plotting the lines as the line parameters are calculated based on the origin being at the upper right corner.

def getSlope(line):
    """
    Returns the slope of a line:
    ((y2-y1)/(x2-x1)) 
    Usefull to decide which segments are part of the left
    line vs. the right line. 
    """
    for x1,y1,x2,y2 in line:
        slope = (y2-y1)/(x2-x1)
        #print(x1,y1,x2,y2,slope)
        return slope

# Adjust image output
plt.rcParams['figure.figsize'] = [12, 4]

# Categorize lines by slope
thresholdSlope = 0; #in degrees
thresholdSlope = math.tan(thresholdSlope*math.pi/180) 

xR = [];yR = [];xL = [];yL = [];
limitLL, limitUL, limitUR, limitLR = vertices[0]


for line in lines:
    x1,y1,x2,y2 = line[0];
    currentSlope = getSlope(line);
    if (currentSlope > thresholdSlope and x1 > limitUL[0] and x2 > limitUL[0]):
    # Its a right line
        if y1>y2:
            x1,x2 = x2,x1
            y1,y2 = y2,y1
        xR += [x1, x2]
        yR += [y1, y2]
    elif (currentSlope < -1.0*thresholdSlope and x1 < limitUR[0] and x2 < limitUR[0]  ):
    # Its a left line
        if y1<y2:
            x1,x2 = x2,x1
            y1,y2 = y2,y1
        xL += [x1, x2]
        yL += [y1, y2]

#print("xR:")
#print(xR)
#print("xL:")
#print(xL)  
    
try:
    zR = np.polyfit(xR, yR, 1)
    mR, bR = zR
    zR = 1/mR , -bR/mR
    fR = np.poly1d(zR)

    for x1, y1 in zip(xR, yR):
        plt.plot(x1, y1, 'bo') 
    plt.plot((fR(limitUR[1]), fR(960)),(limitUR[1], 960), 'b')
except:
    pass

try:
    zL = np.polyfit(xL, yL, 1)
    mL, bL = zL
    zL = 1/mL , -bL/mL
    fL = np.poly1d(zL)

    for x1, y1 in zip(xL, yL):
        plt.plot(x1, y1, 'ro')
    plt.plot((fL(limitUL[1]), fL(960)),(limitUL[1], 960), 'r')
except:
    pass



plt.axis([0, 960, 0, 540])
plt.yticks(rotation=0)
plt.xticks(rotation=0)

# Set origin in upper left
ax=plt.gca()                            # get the axis
ax.set_ylim(ax.get_ylim()[::-1])        # invert the axis


plt.show()

C:\ProgramData\Anaconda3\lib\site-packages\ipykernel_launcher.py:9: RuntimeWarning: divide by zero encountered in int_scalars
  if __name__ == '__main__':

[function] calculateLines

def linesFromCoeffs(z,side,vertices):
    limitLL, limitUL, limitUR, limitLR = vertices[0]
    f = np.poly1d(z)
    if side == "right":
        out = [int(f(limitUR[1])), int(limitUR[1]), int(f(960)),960] 
    else:
        out = [int(f(limitUL[1])),int(limitUL[1]),int(f(960)), 960]
    return out
    
def calculateLines(lines,vertices,thresholdSlope,zL_list=[],zR_list=[]):
    """
    lines is the output of the hough_lines()
    x1R is the x pixel where the right line ends/starts
    x1L is the x pixel where the left line starts/ends
    
    It splits the lines into right lane and left lane lines.
    Then computes the best fit straight that fits all the input lane lines.
    
    It returns the two lines to be drawn by its 4 points
    If it cant compute the lines it returns two zero lenght lines.
    """
    # Categorize lines by slope
    thresholdSlope = math.tan(thresholdSlope*math.pi/180) 

    xR = [];yR = [];xL = [];yL = [];zL =[];zr=[]; 
    global last_mL; global last_bL;
    global last_mR; global last_bR;
    limitLL, limitUL, limitUR, limitLR = vertices[0]
    


    for line in lines:
        x1,y1,x2,y2 = line[0];
        currentSlope = getSlope(line);
        if (currentSlope > thresholdSlope and x1 > limitUL[0] and x2 > limitUL[0]):
        # Its a right line
            if y1>y2:
                x1,x2 = x2,x1
                y1,y2 = y2,y1
            xR += [x1, x2]
            yR += [y1, y2]
        elif (currentSlope < -1*thresholdSlope and x1 < limitUR[0] and x2 < limitUR[0]):
        # Its a left line
            if y1<y2:
                x1,x2 = x2,x1
                y1,y2 = y2,y1
            xL += [x1, x2]
            yL += [y1, y2]
    
    # Parameters for video running mean
    runningAvgCoef = 5 
    maxbDev = 50
    maxmDev = 0.5
    #Right line
    try:
        zR = np.polyfit(xR, yR, 1)
        mR, bR = zR
        zR = 1/mR , -bR/mR
    except:
        zR = last_mR , last_bR
        pass
    try:
        if len(zR_list)<runningAvgCoef:
            last_mR, last_bR = zR;
        
        #Test for variations in b and m
        if abs(abs(last_mR)-abs(zR[0])) < maxmDev and abs(abs(last_bR)-abs(zR[1])) < maxbDev:
            zR_list.append(zR)
        elif len(zR_list)>runningAvgCoef:
                zR = last_mR , last_bR
        else:
            zR = 1/mR , -bR/mR
            last_mR= zR[0]
            last_bR= zR[1]
            zR_list.append(zR)

            
        # Apply moving average for videos
        if len(zR_list)>runningAvgCoef:
            zR_list_short = zR_list[-runningAvgCoef:]
            last_mR = sum([item[0] for item in zR_list_short])/runningAvgCoef
            last_bR = sum([item[1] for item in zR_list_short])/runningAvgCoef
            zR = last_mR , last_bR

        lRCoord = linesFromCoeffs(zR,'r',vertices)
    except Exception as ex:
        
        lRCoord = [0,0, 0, 0]
        print(ex)
        pass

    #Left line
    try:
        zL = np.polyfit(xL, yL, 1)
        mL, bL = zL
        zL = 1/mL , -bL/mL
    except:
        zL = last_mL , last_bL
        pass
    try:
        if len(zL_list)<runningAvgCoef:
            last_mL, last_bL = zL;
        
        #Test for variations in b and m
        if abs(last_mL-zL[0]) < maxmDev and abs(last_bL-zL[1]) < maxbDev:
            zL_list.append(zL)
        elif len(zL_list)>runningAvgCoef:
                zL = last_mL , last_bL
        else:
            zL = 1/mL , -bL/mL
            last_mL= zL[0]
            last_bL= zL[1]
            zL_list.append(zL)

        zL_list.append(zL)
        # Apply moving average for videos
        if len(zL_list)>runningAvgCoef:
            zL_list_short = zL_list[-runningAvgCoef:]
            last_mL = sum([item[0] for item in zL_list_short])/runningAvgCoef
            last_bL = sum([item[1] for item in zL_list_short])/runningAvgCoef
            zL = last_mL , last_bL
        
        lLCoord = linesFromCoeffs(zL,'l',vertices)
    except Exception as ex:
        lLCoord = [0,0, 0, 0]
        print(ex)
        pass


    return [[lLCoord,lRCoord]],zL,zR
        #return [[[x1,y1, x2, y2],
        #        [x1,y1,x2,y2]]]
    

##  Calculate output line lanes

## Test function
imagesLines=[]
titlesLines=[]
imgHouhg,lines = hough_lines(imageROI, rho, theta, threshold, min_line_len, max_line_gap)


if lines.any():
    lines,zL,zR = calculateLines(lines,vertices,thresholdSlope=22)
    #print(lines)
    img_res_lines = np.copy(imgHouhg)*0
    draw_lines(img_res_lines, lines, color=[0, 255, 0], thickness=8)
    
res = weighted_img(img_res_lines, imgHouhg, α=1, β=0.5, γ=0.)

imagesLines.append(imgHouhg);titlesLines.append("Original");
imagesLines.append(res);titlesLines.append("Lines drawn");
    
displayListImages(imagesLines,cols=2,rows=1,titles=titlesLines)

C:\ProgramData\Anaconda3\lib\site-packages\ipykernel_launcher.py:9: RuntimeWarning: divide by zero encountered in int_scalars
  if __name__ == '__main__':

Putting toguether the pipeline

Once all the functions conforming the pipeline has been tested individually, its time to test it all toguether

# Read in the image and print out some stats
images_list = os.listdir("test_images/");

# Load images
image_path = 'test_images/' + images_list[1]
image_base = mpimg.imread(image_path, cv2.IMREAD_COLOR)
print('This image is: ',type(image_base), 
         'with dimensions:', image_base.shape)

###################### Pipeline start #################
pipImages = []
piptitles = []

###### Resize image
image_base = resizeImage(image_base)
imagesCE.append(image_base);titlesCE.append("Original");

###### Color Enhancement
imageCE = colorEnhancement(image_base)
pipImages.append(imageCE);piptitles.append("Color Enhanced");

###### GrayScale
imageGray = grayscale(imageCE)
pipImages.append(imageGray);piptitles.append("Image Gray");

###### Gauss Smoothing
imageGauss = gaussian_blur(imageGray,5)
pipImages.append(imageGauss);piptitles.append("Image Gauss");

###### Canny Edge
imageCanny = cannyEdge(imageGauss,250,450)
pipImages.append(imageCanny);piptitles.append("Canny Edge");

###### ROI
ysize =imageCanny.shape[0]
xsize =imageCanny.shape[1]

ROI_upperWidth = 100 #Width of the upper horizontal straight in px
ROI_upperHeight = 200 #Height of the upper horizontal straight from the bottom of the image in px
ROI_lowerWidth = 750 #Width of the lower horizontal straight in px
ROI_lowerHeight = 50  #Height of the lower horizontal straight  from the bottom of the image in px      
    
limitLL = ((xsize/2)-(ROI_lowerWidth/2),ysize-ROI_lowerHeight);
limitLR = (xsize - ((xsize/2)-(ROI_lowerWidth/2)),ysize-ROI_lowerHeight);
limitUL = ((xsize/2)-(ROI_upperWidth/2), ysize-ROI_upperHeight);
limitUR = ((xsize/2)+(ROI_upperWidth/2), ysize-ROI_upperHeight);
vertices = np.array([[limitLL,limitUL,limitUR , limitLR]], dtype=np.int32)

imageROI = region_of_interest(imageCanny,vertices)
pipImages.append(imageROI);piptitles.append("ROI");

###### Hough lines
rho = 1 # distance resolution in pixels of the Hough grid
theta = 1*np.pi/180 # angular resolution in radians of the Hough grid
threshold = 15    # minimum number of votes (intersections in Hough grid cell)
min_line_len = 5 #minimum number of pixels making up a line
max_line_gap = 40   # maximum gap in pixels between connectable line segments

imgHouhg,lines = hough_lines(imageROI, rho, theta, threshold, min_line_len, max_line_gap)

pipImages.append(imgHouhg);piptitles.append("Houhg lines");

###### Lane lines
if lines.any():
    linesArrow,zL,zR = calculateLines(lines,vertices,thresholdSlope=22)
    #print(lines)
    img_res_lines = np.copy(imgHouhg)*0
    draw_lines(img_res_lines, linesArrow, color=[0, 255, 0], thickness=8)
    
res = weighted_img(img_res_lines, image_base, α=1, β=0.5, γ=0.)
pipImages.append(res);piptitles.append("Result");
    
displayListImages(pipImages,cols=2,rows=4,titles=piptitles)

This image is:  <class 'numpy.ndarray'> with dimensions: (720, 1280, 3)

[function] process_image

That is it basically, once the pipeline has been correctly tuned, it can be applied to the original images to see the ressults

def process_image(img,canny_low=50,canny_high=150,
                          ROI_upperWidth=120,
                          ROI_upperHeight=210,
                          ROI_lowerWidth=750,
                          ROI_lowerHeight=50,
                          GSKernel =7,
                          hough_rho = 1,
                          hough_theta = 1,
                          hough_threshodl = 10,
                          hough_min_line_len  = 5,
                          hough_max_line_gap  = 10,
                          thresholdSlope=20,
                          plots = False,
                          video= True
                         ):
    #### Frame dependence for video
    if not video:
            global zL_list
            zL_list= []
            global zR_list
            zR_list= []

    ###################### Pipeline start #################
    
    ###### Resize image
    img = resizeImage(img)
    
    ###### Color Enhancement
    imageCE = colorEnhancement(img)

    ###### GrayScale
    imageGray = grayscale(imageCE)

    ###### Gauss Smoothing
    imageGauss = gaussian_blur(imageGray,GSKernel)

    ###### Canny Edge 
    imageCanny = cannyEdge(imageGauss,canny_low,canny_high)
    
    ###### ROI
    ysize =imageCanny.shape[0]
    xsize =imageCanny.shape[1]
 
    limitLL = ((xsize/2)-(ROI_lowerWidth/2),ysize-ROI_lowerHeight);
    limitLR = (xsize - ((xsize/2)-(ROI_lowerWidth/2)),ysize-ROI_lowerHeight);
    limitUL = ((xsize/2)-(ROI_upperWidth/2), ysize-ROI_upperHeight);
    limitUR = ((xsize/2)+(ROI_upperWidth/2), ysize-ROI_upperHeight);
    vertices = np.array([[limitLL,limitUL,limitUR , limitLR]], dtype=np.int32)
    
    imageROI= region_of_interest(imageCanny, vertices)

    # Plot vertices for verification in original image
    x = [limitLL[0], limitUL[0], limitUR[0], limitLR[0],limitLL[0]]
    y = [limitLL[1], limitUL[1], limitUR[1], limitLR[1],limitLL[1]]

    ###### Hough lines
    theta = hough_theta*np.pi/180 # angular resolution in radians of the Hough grid

    imgHouhg,lines = hough_lines(imageROI, hough_rho, hough_theta, hough_threshodl, hough_min_line_len, hough_max_line_gap,)
    imgHouhg = weighted_img(imgHouhg, img, α=1, β=0.9, γ=0.)

    ###### Lane lines
    if len(lines)>0:
        linesArrow,zL,zR = calculateLines(lines,vertices,thresholdSlope,zL_list,zR_list)
        #print(lines)
        res = np.copy(img)*0
        draw_lines(res, linesArrow, color=[0, 255, 0], thickness=8)
        #imgHouhg = weighted_img(res, imgHouhg, α=1, β=0.5, γ=0.)
        image_out = weighted_img(res, img, α=1, β=0.5, γ=0.)
    else:
        image_out = img
    
    ###### If plots are required
    if plots:
        # Helper to adapt images to full width
        plt.rcParams['figure.figsize'] = [12, 8]
        plt.rcParams['figure.dpi'] = 100 # 200 e.g. is really fine, but slower

        # Helper function to display images correctly
        fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2)
        ax1.imshow(imageCE, cmap=None)
        ax2.imshow(imageCanny, cmap=None)
        ax1.plot(x, y, 'b--', lw=4)
        ax3.imshow(imgHouhg)
        ax4.imshow(image_out)
        return
       
    return image_out

#Test function
a = process_image(images[0],plots=False, video=False)
plt.figure()
plt.imshow(a, cmap=None)
# or 
process_image(images[0],plots=True,video=False)

Tune pipeline parameters

Interactive function to tune parameters to optimize line detection.

from ipywidgets import *
from IPython.display import display

# Read in the image and print out some stats
images_list = os.listdir("test_images/");
# Load images
image_path = 'test_images/' + images_list[1]
image_root = mpimg.imread(image_path, cv2.IMREAD_COLOR)


#Define sliders
canny_low_slider = IntSlider(value=250,min=0,max=500,step=1,continuous_update=False,)
canny_high_slider = IntSlider(value=450,min=0,max=1000,step=1,continuous_update=False,)
GSKernel_slider = IntSlider(value=5,min=1,max=55,step=2,continuous_update=False,)
hough_rho_slider = FloatSlider(value=1.0,min=0.1,max=10,step=0.1,continuous_update=False,)
hough_theta_slider = FloatSlider(value=1.0,min=0.1,max=10,step=0.1,continuous_update=False,)
hough_threshodl_slider = IntSlider(value=15,min=0,max=100,step=1,continuous_update=False,)
hough_min_line_len_slider = IntSlider(value=5,min=0,max=400,step=1,continuous_update=False,)
hough_max_line_gap_slider = IntSlider(value=40,min=0,max=400,step=1,continuous_update=False,)
thresholdSlope_slider = IntSlider(value=0,min=0,max=90,step=1,continuous_update=False,)


#Create interactive function
def interactive(canny_low,
                canny_high,
                GSKernel,
                hough_rho,
                hough_theta,
                hough_threshodl,
                hough_min_line_len,
                hough_max_line_gap,
                thresholdSlope):
    return process_image(image_root,
                                 canny_low=canny_low,
                                 canny_high=canny_high,
                                 GSKernel=GSKernel,
                                 hough_rho=hough_rho,
                                 hough_theta=hough_theta,
                                 hough_threshodl=hough_threshodl,
                                 hough_min_line_len=hough_min_line_len,
                                 hough_max_line_gap=hough_max_line_gap,
                                thresholdSlope=thresholdSlope,
                                plots = True,
                                video=False,)
# Run interacctive function
interact(interactive, canny_low=canny_low_slider,
         canny_high=canny_high_slider,
         GSKernel=GSKernel_slider,
        hough_rho=hough_rho_slider,
        hough_theta=hough_theta_slider,
         hough_threshodl=hough_threshodl_slider,
         hough_min_line_len=hough_min_line_len_slider,
         hough_max_line_gap=hough_max_line_gap_slider,
         thresholdSlope = thresholdSlope_slider,
        )

<function __main__.interactive(canny_low, canny_high, GSKernel, hough_rho, hough_theta, hough_threshodl, hough_min_line_len, hough_max_line_gap, thresholdSlope)>

Results

Results applied to the initial images

# Read list of images from a folder and obtain path
images_path ='test_images/'
names = os.listdir(images_path)
images_list = [images_path + item for item in names]
#Create matplotlib images from a list of paths
images =  [mpimg.imread(item, cv2.IMREAD_COLOR) for item in images_list]; 

results = []

for i in range(len(images)):
    res = process_image(images[i], plots = False,video=False)
    results.append(res)
    mpimg.imsave('test_images_output/' + names[i],res)
    
displayListImages(results,cols=2,rows=6,titles=names)

Results of test on images

The results are quite good it generally tracks the lines correctly (if they are vissible). There is a bit of wobling around dashed lines as dots can be ignored sometimes derivating in a slight change in angle.

Overall the pipeline works as expected detecting white and yellow line markigs and sucessfully filtering out most of the background noise, even with shadows and tarmac changes

Test on Videos

# Import everything needed to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML

Smothing lines on videos

# Necessary to reset previous frame properties to carry out moving average smoothing
global zR_list 
zR_list = []
global zL_list 
zL_list = []

white_output = 'test_videos_output/solidWhiteRight.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
#clip1 = VideoFileClip("test_videos/solidWhiteRight.mp4").subclip(0,2)
clip1 = VideoFileClip("test_videos/solidWhiteRight.mp4")
white_clip = clip1.fl_image(process_image) #NOTE: this function expects color images!!
%time white_clip.write_videofile(white_output, audio=False)

Moviepy - Building video test_videos_output/solidWhiteRight.mp4.
Moviepy - Writing video test_videos_output/solidWhiteRight.mp4

Moviepy - Done !
Moviepy - video ready test_videos_output/solidWhiteRight.mp4
Wall time: 6.67 s

Play the video inline, or if you prefer find the video in your filesystem (should be in the same directory) and play it in your video player of choice.

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(white_output))

Now for the one with the solid yellow lane on the left. This one's more tricky!

# Necessary to reset previous frame properties
global zR_list 
zR_list = []
global zL_list 
zL_list = []

yellow_output = 'test_videos_output/solidYellowLeft.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
#clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4').subclip(0,5)
clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4')
yellow_clip = clip2.fl_image(process_image)
%time yellow_clip.write_videofile(yellow_output, audio=False)

Moviepy - Building video test_videos_output/solidYellowLeft.mp4.
Moviepy - Writing video test_videos_output/solidYellowLeft.mp4

Moviepy - Done !
Moviepy - video ready test_videos_output/solidYellowLeft.mp4
Wall time: 19.3 s

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(yellow_output))

Writeup and Submission

If you're satisfied with your video outputs, it's time to make the report writeup in a pdf or markdown file. Once you have this Ipython notebook ready along with the writeup, it's time to submit for review! Here is a link to the writeup template file.

Optional Challenge

Try your lane finding pipeline on the video below. Does it still work? Can you figure out a way to make it more robust? If you're up for the challenge, modify your pipeline so it works with this video and submit it along with the rest of your project!

# Necessary to reset previous frame properties
global zR_list 
zR_list = []
global zL_list 
zL_list = []

challenge_output = 'test_videos_output/challenge.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
##clip3 = VideoFileClip('test_videos/challenge.mp4').subclip(0,2)

clip3 = VideoFileClip('test_videos/challenge.mp4')
challenge_clip = clip3.fl_image(process_image)
%time challenge_clip.write_videofile(challenge_output, audio=False)

Moviepy - Building video test_videos_output/challenge.mp4.
Moviepy - Writing video test_videos_output/challenge.mp4

Moviepy - Done !
Moviepy - video ready test_videos_output/challenge.mp4
Wall time: 8.88 s

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(challenge_output))

Improvements:

• Improve image processing specially of shadows
• Left line tends to tighten
• Left line moves more than right
• Lines have differnt height
• Unify resolutions
• Draw lines individually if availiable
• HSL pre filtering
  • Yellow color isolation
  • White color isolation
• Increase Smoothing in videos by taking into account previous frames
  • Smooth out behaviour dotted lines
• Apply statistics in hough function to detect odd straights
• Improve impelmentation of video frame smoothing
  • Try objects and classes?
• Why sometimes the polyfit does not fidn a straight even there are suitable hough lines

One-off helper functions

Function to save images from videos

clip3.save_frame("test_videos/challenge_7s.png", t=7) # saves the frame a t=xs

yellow_clip.write_gif("test_videos_output/solidYellowLeft.gif",fps=15)
white_clip.write_gif("test_videos_output/solidWhiteRight.gif",fps=15)
challenge_clip.write_gif("test_videos_output/challenge_.gif",fps=15)

OpenCV functions that might be helpfull

cv2.inRange() for color selection
cv2.fillPoly() for regions selection
cv2.line() to draw lines on an image given endpoints
cv2.addWeighted() to coadd / overlay two images
cv2.cvtColor() to grayscale or change color
cv2.imwrite() to output images to file
cv2.bitwise_and() to apply a mask to an image